Could you help me understand why we need the unsafe_hard_delete changes? Its not really clear to me what behavior this enables which we can't already do

It's mainly because the way that models are passed around is directly a _SingletonProxy instead of _SingletonEntry so we would need a way to directly call delete with the _SingletonProxy

Ok - lets at least give it a name like singletonProxy_unsafe_hard_delete. Otherwise we will run into issues if someone has an object with a function or property called unsafe_hard_delete, which seems like it could happen.

Sg! Updated.

I assume this is so that this is pickleable, but is it valid? Normally I'd expect this to not be pickleable since the proxy objects aren't necessarily valid in another context

Yeah this is exactly what was needed for the pickling stuff. It does seems to be valid in testing with the custom built beam version loaded on custom container.

I think it would only be valid if you unpickle onto the same machine (and maybe even in the same process). Could you remind me what unpickling issues you ran into?

Just tried removing these and run the test locally, it's this infinite recursion thing that will happen if i have a proxy on a proxy

<string>:2: in make_proxy
    ???
../../../../.pyenv/versions/3.11.14/lib/python3.11/multiprocessing/managers.py:822: in _callmethod
    kind, result = conn.recv()
                   ^^^^^^^^^^^
../../../../.pyenv/versions/3.11.14/lib/python3.11/multiprocessing/connection.py:251: in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apache_beam/utils/multi_process_shared.py:226: in __getattr__
    return getattr(self._proxyObject, name)
                   ^^^^^^^^^^^^^^^^^
apache_beam/utils/multi_process_shared.py:226: in __getattr__
    return getattr(self._proxyObject, name)
                   ^^^^^^^^^^^^^^^^^
apache_beam/utils/multi_process_shared.py:226: in __getattr__
    return getattr(self._proxyObject, name)
                   ^^^^^^^^^^^^^^^^^
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)

by proxy on proxy I meant that first a MultiProcessShared object is created and the instance initialized inside it also try to create multiprocessshared objects. So for example like this test

class SimpleClass:
  def make_proxy(
      self, tag: str = 'proxy_on_proxy', spawn_process: bool = False):
    return multi_process_shared.MultiProcessShared(
        Counter, tag=tag, always_proxy=True,
        spawn_process=spawn_process).acquire()

def test_proxy_on_proxy(self):
    shared1 = multi_process_shared.MultiProcessShared(
        SimpleClass, tag='proxy_on_proxy_main', always_proxy=True)
    instance = shared1.acquire()
    proxy_instance = instance.make_proxy()
    self.assertEqual(proxy_instance.increment(), 1)

Why are we sending a kill signal to the parent process? Isn't this the opposite of what we want?

This is not actually a kill signal but uses that interface to send a check, it will fail with OSError if the parent_pid is dead and if alive nothing happens.

if alive nothing happens.

Could you help me understand why this happens? https://www.geeksforgeeks.org/python/python-os-kill-method/ seems to say this will actually send the kill signal. Does the parent just ignore it?

This might be a good description https://man7.org/linux/man-pages/man2/kill.2.html

In short if signal 0 is passed in it actually doesn't send the kill signal and instead just do existence and permission checks.